Image acquisition method for time of flight camera

ABSTRACT

A method of reduce the impact of noise on a depth image produced using a Time Of Flight (TOF) camera uses an infrared image produced from one or more phase-specific images captured by the TOF camera to determine whether to move pixels in the depth image from one phase section to another.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority under 35 U.S.C. § 119 to Korean PatentApplication No. 10-2021-0090674 filed on Jul. 12, 2021, which isincorporated herein by reference in its entirety.

BACKGROUND 1. Field

The present disclosure relates to a technology for reducing noise in animage captured by a time of flight (TOF) camera. The present disclosurerelates to a technology of reducing noise by correcting an error whichoccurs when unwrapping a TOF image using multiple frequencies to producea depth image, through statistical analysis with neighboring pixels, anddetermining depth information to be similar between similar pixels inorder to optimize the depth image.

2. Discussion of the Related Art

Recently, with the development of a semiconductor technology, functionsof Information Technology (IT) devices with built-in cameras have becomemore diversified. Technologies of acquiring and processing images usingcharge coupled devices (CCD) or CMOS image sensor (CIS) devices havealso been rapidly developed.

Among technologies for acquiring three-dimensional images,time-of-flight (TOF) techniques have been widely used recently. The TOFrefers to a process of measuring a time required for light emitted froma light source and reflected from a subject to returns to the lightsource, calculating a distance between the light source and the subjectaccording to that time, and thereby acquiring a depth image. Among theTOF techniques, a process of calculating a distance by a time differencebetween the time when light departed from a light source and the timewhen the light reflected from a subject reached a sensor located nearthe light source is called a direct TOF method. Since the speed of lightis constant, it is possible to easily and simply calculate a distancebetween a light source and a subject by finding the time difference.However, this method has the following disadvantages. Since lighttravels 30 centimeters in one nanosecond, a resolution of picosecondsmay be required in order to acquire a usable depth image of the subject.However, it is somewhat difficult to provide the resolution ofpicoseconds at the current operating speed of semiconductor devices.

In order to solve such a problem, an indirect TOF (I-TOF) process hasbeen developed. This is a method in which light is modulated with asignal having a specific frequency for transmission, and when themodulated signal reflected from a subject has reached a sensor, a phasedifference between the modulated and received signals is detected tocalculate the distance to the subject. The modulation process may use apulse signal or a sine wave. In the case of using a pulse signal, aprocess of acquiring a necessary image by measuring signals at differenttime periods with signals having four different phases and thenunwrapping a depth image using phase values is widely used. FIG. 1 andFIG. 2 are diagrams for explaining such a process. Four imagesrespectively acquired using the four different phases are labeled Q1 toQ4, respectively. In each of the upper four images shown in FIG. 2 , anintensity of a pixel corresponds to a degree of overlap between thepulse in the reflected wave and a pulse in the corresponding phase,wherein the degree of overlap corresponds to the shaded regions in FIG.1 . It is well known that a depth image from the four images may berestored using a phase restoration function with an arc tangent, such asthe phase restoration function shown in FIG. 2 .

However, the I-TOF has a problem in that it is difficult to measure anaccurate depth value when the range of a depth image exceeds thedistance light travels during a period of a pulse signal, and a distanceto a subject is limited. The related art for solving such a problem is amethod of using phases having two or more frequencies to acquire aplurality of images. When images acquired with phases of two frequenciesare combined, long-distance images may also be properly restored, butwhen the long-distance images are unwrapped from the acquired images,noise may be involved, which may produce an error and affect the qualityof the restored images.

RELATED ART DOCUMENT

[Patent Document]

-   Patent Document 1: US Patent Application Publication No.    2020/0333467 published on Oct. 22, 2020.-   Patent Document 2: Korean Patent Application Publication No.    10-2013-0099735 published on Sep. 6, 2013.

SUMMARY

The technical problem addressed by the present disclosure is to providea process of correcting a depth image so that an error caused by noiseis removed when a TOF camera is used.

Another technical problem addressed by the present disclosure is toprovide a process for estimating an error value for each pixel when adepth image of a subject is acquired using a TOF camera, and using theestimated error value in securing image quality.

Another technical problem addressed by the present disclosure is toprovide a process for improving the quality of a depth image of asubject by acquiring an error value for each pixel using a mathematicalprocess when the depth image of the subject is acquired using a TOFcamera.

An embodiment of the present disclosure for solving the above problemsis an image acquisition method for a TOF camera, which includes: a stepof acquiring an image with a first frequency and an image with a secondfrequency; a step of acquiring an unwrapped image based on the acquiredimages; a step of selecting a partial area from the unwrapped image; astep of distinguishing some pixels among pixels belonging to the partialarea from other pixels; and a step of moving a phase section to whicherroneously unwrapped pixels belong to another phase section.

Another embodiment of the present disclosure for solving the aboveproblems is an image acquisition method for a TOF camera, whichincludes: a step of acquiring an initial image from a TOF camera; a stepof unwrapping the initial image; a step of acquiring an infrared imagefrom the initial image; a step of filtering the initial image based ondispersion; a step of correcting the unwrapped initial image; and a stepof performing denoising based on the infrared image, an image acquiredby the filtering, and an image acquired in the step of correcting.

According to the present disclosure, it is possible to secure along-distance image, which is relatively unconstrained by a distance toa subject, by a TOF camera and to minimize an influence of noise thatmay be involved in the unwrapping of the image. Consequently, it ispossible to achieve a high-quality restored image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an operation principle of a camera using an indirectTOF method.

FIG. 2 illustrates a principle of restoring an image captured by a TOFcamera.

FIG. 3 illustrates a process of restoring an unwrapped image from imagesacquired using a plurality of frequencies.

FIG. 4 shows a pixel area showing an error due to noise involved in aprocess of restoring an image by unwrapping according to an embodiment.

FIG. 5 illustrates a process for correcting unwrapping by skipping oneperiod of a frequency.

FIG. 6 illustrates an erroneously unwrapped pixel due to involved noise.

FIG. 7 is a flowchart of a process of moving a pixel section bycorrection unwrapping according to an embodiment.

FIG. 8 illustrates a pixel area where a lot of noise is involved.

FIG. 9 is a flowchart of a process of properly denoising a pixel areawhere a lot of noise is involved according to an embodiment.

FIG. 10 is illustrates a reliability evaluation image according to anembodiment.

FIG. 11 is a step-by-step view of images derived during a denoisingprocess according to an embodiment.

DETAILED DESCRIPTION

Hereinafter, embodiments of the present disclosure will be described indetail with reference to the accompanying drawings such that the presentdisclosure can be easily carried out by those skilled in the art towhich the present disclosure pertains. The same reference numerals amongthe reference numerals in each drawing indicate the same members.

In the description of the present disclosure, when it is determined thatdetailed descriptions of related publicly-known technologies may obscurethe subject matter of the present disclosure, the detailed descriptionsthereof will be omitted.

The terms such as first and second may be used to describe variouscomponents, but the components are not limited by the terms, and theterms are used only to distinguish one component from another component.

Hereinafter, the present disclosure will be described with reference tothe related drawings.

A process of acquiring an unwrapped image from multi-frequency images asillustrated in FIG. 3 will be described as an example. When a pulsesignal with multiple frequencies, for example, a first frequency of80.32 MHz and a second frequency of 60.24 MHz, is used, phases of thefirst frequency and the second frequency coincide with each other againafter four periods and three periods from the same start point, and thisbehavior is repeated. Hereinafter, for convenience of description, sucha period in which multiple frequencies are repeated is referred to as a‘repetition period’ and is equal to the inverse of the greatest commondivisor (GCD) of the first and second frequencies. Here, that GCD is20.08 MHz, and accordingly in the example of FIG. 3 , 4.98 nanosecondsis the repetition period. Since the first frequency and the secondfrequency are pulses having different phases, the image may be dividedinto sections in which phases are relatively changed within onerepetition period. For example, in FIG. 3 , wherein a dashed line in thegraph indicates a distance measurement produced using the firstfrequency of 80.32 MHz and a solid line in the graph indicates adistance measurement produced using the second frequency of 60.24 MHz,section {circle around (1)} is a section in which the first frequencyand the second frequency overlap each other (that is, have identicalphase angles and therefore produce images with identical distancevalues), section {circle around (2)} is a section in which the firstfrequency transitions into a new cycle first and therefore the firstfrequency produces a distance much less than the distance produced bythe second frequency, section {circle around (3)} is a section in whichthe second frequency transitions into a new cycle subsequent to thetransition into a new cycle of the first frequency and therefore thefirst frequency produces a distance greater than the distance producedby the second frequency, and section {circle around (4)}, section{circle around (5)}, and section {circle around (6)}) may be describedin a similar way.

Therefore, in FIG. 3 , an image with the first frequency is an imageacquired during the repetition period, that is, during which the pulseof 80.32 MHz is repeated four times, and an image with the secondfrequency is an image acquired during which the frequency of 60.24 MHzis repeated three times. When one repetition period (here, 4.98nanoseconds) is converted into a distance that can be measured withoutambiguity by multiplying it by one-half the speed of light, the distanceis 7.46 meters. For the first frequency of 80.32 MHz, a distancecorresponding to one period is 1.87 meters, and for the second frequencyof 60.24 MHz, a distance corresponding to one period is 2.49 meters.

When frequency information is properly used, the image acquired usingthe first frequency and the image acquired using the second frequencyduring the repetition period, or an image overlapped by a combinationthereof may be restored through proper unwrapping, and the resultingimage is referred to as an unwrapped image as illustrated in FIG. 3 . Inother words, for the two frequencies used in this example, six sectionscorrespond to six cases, and from the information in the images acquiredusing the two frequencies, it is possible to know which of the cases aspecific pixel corresponds to, so that it becomes possible to restore aproper image having proper distances by unwrapping an image.

However, there is a possibility that a pixel error will occur due to aproblem such as noise introduced when an unwrapping operation isperformed, noise in the image acquired using the first frequency, ornoise in the image acquired using the second frequency. An example ofFIG. 4 illustrates a pixel error that occurs when pixel data that in theabsence of the error would belong to section {circle around (3)} isinstead determined to belong to a section before or after one period dueto the error. That is, the example of FIG. 4 illustrates a result inwhich a difference of 2 πN, which is an integer multiple of one period,occurs due to erroneous unwrapping of the pixel data. Therefore, in anembodiment, if a first unwrapping (referred to as an ‘initialunwrapping’) is wrong, when the pixel data is adjusted by 2 πN throughcomparison with surrounding values, a corrected unwrapping (referred toas a ‘correction unwrapping’) result illustrated in FIG. 5 may beacquired.

One embodiment of a process of performing correction unwrapping will bedescribed with reference to steps illustrated in FIG. 7 . First, animage produced using a first frequency and an image produced using asecond frequency are acquired with a TOF camera (steps S10 and S11). Anunwrapped image is acquired with these images (step S20). Pixels in acertain area including erroneously unwrapped pixels, for example, pixelsof a 3×3 patch are selected from the unwrapped image (step S30). Next,when an unwrapping section of the pixels belonging to the area isexpressed using a mathematical tool, for example, a histogram, thepixels are divided into properly unwrapped pixels and erroneouslyunwrapped pixels as illustrated in FIG. 6 (step S40). Most of the pixelsin the patch are unwrapped to belong to section {circle around (3)}, butsome pixels are unwrapped to belong to section {circle around (5)},which is taken as an indication that the pixels in section {circlearound (5)} are erroneously unwrapped pixels. Then, the erroneouslyunwrapped pixels are compared with adjacent pixels and it is determinedwhether to move the erroneously unwrapped pixels to a section to whichmost of the adjacent pixels belong (step S50). The erroneously unwrappedpixels are moved to section {circle around (3)} according to thecomparison result (step S60). The movement is effectively performedaccording to comparison of the values of the erroneously unwrappedpixels with surrounding values. At this time, it is preferable to use amode filter. In such a case, a 3×3 mode filter for coping with the 3×3patch is suitable. In this way, a corrected unwrapping result may beacquired. This series of steps are illustrated in the flowchart of FIG.7 .

In another embodiment, an area including a lot of noise in an unwrappedimage is corrected. For example, it is preferable to correct an areaincluding a lot of noise, such as an area illustrated in FIG. 8 , byusing a mode filter, or in another embodiment without using the modefilter. As illustrated in the histogram of FIG. 8 , a properly unwrappedvalue belongs to section {circle around (2)}, but is less prominent thanvalues erroneously unwrapped and dispersed into other sections due toinvolved noise. That is, in this example, a dispersion value is large.When the dispersion value is large, it means that the reliability of animage is reduced. Due to such characteristics, correction throughcomparison with neighboring pixels becomes more difficult. When such acase occurs, embodiments may remove noise based on an infrared intensityvalue calculated by the TOF camera. This process is based on theprinciple that when the pixel intensity value of an erroneouslyunwrapped pixel is similar to that of a neighboring pixel, they arehighly likely to be the same subject, and therefore depth values thereofare also highly likely to be similar.

Another embodiment of the present disclosure will be described below inmore detail with reference to FIG. 9 . An initial image is acquired fromthe TOF camera (step S110). In order to acquire the initial image, acalculation process using a phase restoration algorithm may be used asneeded. An image is acquired by unwrapping the initial image (stepS120), and an unwrapped image is acquired by correcting the image (stepS130). A process of acquiring the corrected unwrapped image may be anembodiment of the present disclosure described above. An image, whosereliability may be evaluated through dispersion-based filtering, isacquired from the initial image (step S220). In such a case, it ispreferable to calculate a dispersion value as a reliability value or tocalculate a dispersion value by using appropriate scaling. In thereliability evaluation, it is necessary to set the reliability value ofan area including a lot of noise to be small. It is empirically knownthat a lot of noise is involved when a subject surface is black or theorientation direction of the subject surface is changing. For reference,since the pixel value of an area including a lot of involved noisechanges significantly, a dispersion value thereof is also increased.When a dispersion value is low, a high reliability value is set. This iswell described in the drawing of FIG. 10 . FIG. 10 illustrates anexample of applying a process in which the reliability of an areaincluding a lot of noise is set to be low based on a dispersion value inthe image with the first frequency of 80.32 MHz. Particularly, FIG. 10illustrates a case where a lot of noise is involved in an area where theorientation direction of a subject surface changes.

Next, an infrared intensity image is separately acquired from theinitial image (step S320). The infrared intensity image refers toamplitude information determined using the amplitude information of theinitial image data for all of the phases; for example, in an embodiment,the four images respectively corresponding to phases Q1, Q2, Q3, and Q4shown in FIG. 2 may be combined by summation to produce the infraredintensity image. For reference, the aforementioned reliability imageuses a phase value or a frequency value of the initial image, whereasthe infrared image uses the amplitude information. By performing adenoising step (S140) of removing noise from the corrected unwrappedimage using the dispersion-based reliability image and the infraredintensity image, a resultant image is acquired. FIG. 11 illustratesimages acquired in each step for reference.

In the denoising step (S140), the intensity value of the infraredintensity image is used as a guide image for reference. Furthermore, inthe processing of the initial image acquired from the TOF camera or thedenoising step, a well-known algorithm such as a Bilateral Solver may beused or other algorithms may also be used. When the Bilateral Solver isapplied, a denoised depth image may be acquired by substituting pixelvalues of a high-reliability surrounding area for pixel values of alow-reliability surrounding area, using the guide image and thereliability area to determine when to perform such substitutions.

Although the present disclosure has been described with reference to theembodiments illustrated in the drawings, these are for illustrativepurposes only, and those skilled in the art will appreciate that variousmodifications and other equivalent embodiments are possible from theembodiments. Thus, the true technical scope of the present disclosureshould be defined according to the appended claims.

What is claimed is:
 1. An image acquisition method for a Time Of Flight(TOF) camera, the image acquisition method comprising: acquiring a firstimage using a first frequency; acquiring a second image using a secondfrequency; producing an initial unwrapped image based on the first andsecond images; selecting a partial area from the unwrapped image;distinguishing one or more erroneously unwrapped pixels of the partialarea from remaining pixels of the partial area; and moving anerroneously unwrapped pixel from a phase section to which it has beeninitially unwrapped to another phase section.
 2. The image acquisitionmethod of claim 1, further comprising: processing the partial area usinga mode filter before moving the erroneously unwrapped pixel.
 3. Theimage acquisition method of claim 1, wherein distinguishing the one ormore erroneously unwrapped pixels includes comparing the one or moreerroneously unwrapped pixels with the remaining pixels.
 4. The imageacquisition method of claim 1, wherein distinguishing the one or moreerroneously unwrapped pixels includes performing a mathematical processfor analyzing unwrapped values of pixels belonging to the partial area.5. The image acquisition method of claim 4, wherein the mathematicalprocess includes creating a histogram.
 6. The image acquisition methodof claim 4, wherein the mathematical process includes determining adispersion value.
 7. The image acquisition method of claim 4, whereinthe mathematical process is for distinguishing the erroneously unwrappedpixels.
 8. The image acquisition method of claim 1, wherein moving theerroneously unwrapped pixel is based on a value selected from one periodof the first frequency and one period of the second frequency.
 9. Animage acquisition method for a TOF camera, the image acquisition methodcomprising: acquiring an initial image from a TOF camera; unwrapping theinitial image to produce an unwrapped initial image; acquiring aninfrared image from the initial image; filtering the initial image toproduce a reliability evaluation image; correcting the unwrapped initialimage to produce a corrected unwrapped image; and performing denoisingbased on the infrared image, the reliability evaluation image, and thecorrected unwrapped image.
 10. The image acquisition method of claim 9,wherein the correcting the unwrapped initial image includes comparingsome pixels with remaining pixels.
 11. The image acquisition method ofclaim 9, wherein the correcting the unwrapped initial image includesperforming a mathematical process for analyzing unwrapped values of somepixels.
 12. The image acquisition method of claim 11, wherein themathematical process includes creating a histogram.
 13. The imageacquisition method of claim 11, wherein the mathematical processincludes determining a dispersion value.
 14. The image acquisitionmethod of claim 9, wherein performing denoising includes using theinfrared image as a reference image.
 15. The image acquisition method ofclaim 9, wherein filtering the initial image is based on a dispersionvalue among pieces of information of the initial image.
 16. The imageacquisition method of claim 9, wherein the reliability evaluation imageis used for evaluating reliability of the initial image.
 17. The imageacquisition method of claim 16, wherein when evaluating reliability ofthe initial image, more noise in the initial image corresponds to alower evaluation.
 18. The image acquisition method of claim 9, whereinthe infrared image is acquired using amplitude information among piecesof information of the initial image.